Intelligent interface based on augmented reality

ABSTRACT

An interaction method implemented by a terminal apparatus includes: obtaining an interface underlayer of a space that has direction information and distance information; establishing a mapping relationship between an external device and an identification corresponding to the external device marked on the interface underlayer, the identification indicating a physical location of the external device; receiving, from the external device, property information of the external device; obtaining an image of a real scene in the space; determining that a target object is depicted in the image of the real scene; determining target presentation information of the target object according to the property information of the external device; and displaying the image of the real scene together with the target presentation information, the target presentation information being added to the image as an augmented reality (AR) mark at an image location corresponding to the target object.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation-in-part application of U.S. patentapplication Ser. No. 15/547,939. U.S. patent application Ser. No.15/547,939 is a national stage application of PCT Patent Application No.PCT/CN2016/070454, filed on Jan. 8, 2016, which claims priority toChinese Patent Application No. 201610008470.1, filed on Jan. 7, 2016,the entire content of all the above-mentioned applications areincorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to the field of interactive control, andmore particularly to an intelligent interface.

BACKGROUND

Currently, using a ranging device to measure and plan routes and areasis required almost in all aspects of people's life and work, such as inthe field of construction, road engineering, transportation, pipelinelaying, garden landscape, etc. Commonly used ranging devices are laserrangefinder and ranging wheel.

Various ranging device structures have been disclosed in prior art. Forexample, CN201420143325.0 discloses an inner and outer optical pathswitching system of a laser rangefinder for realizing switch of thelaser rangefinder in two working status of an inner optical path and anouter optical path. The switching system includes a laser source, a halftransmitting half-reflecting mirror, a reflecting mirror, a shadingplate, a motor and a motor-driven circuit. The shading plate is fixed ona rotating shaft of the motor and is driven by the motor to select to beat a first shading position or a second shading position. The laseremitted by the laser source is divided into two light waves of ranginglight and inner reference light through the half-transmittinghalf-reflecting mirror. The ranging light penetrates through thehalf-transmitting half-reflecting mirror and forms the outer opticalpath. The inner reference light is reflected by the reflecting mirrorand forms the inner optical path. For another example, CN201410152898.4discloses a single optical path laser ranging system for accuratedetermination of the distance of measured object. The ranging systemcomprises a microprocessor, a signal generator, a laser generator, and aphotodiode and frequency mixer. The microprocessor controls the signalgenerator to output two groups of pulse signals. One group of pulsesignals are respectively sent to the laser generator and thephotodiodes, to form a photoelectric mixed signal used by the outeroptical path. Another group of pulse signals are sent to the frequencymixer for frequency mixing to generate a reference signal used byranging. The microprocessor synchronously samples the photoelectricmixed signal and the reference signal, and calculates the voltage valuesof both signals to accurately obtain a measured distance value.

However, it can be seen, from the technical solutions disclosed above,that conventional ranging device can only perform straight line ranging,none of them is capable of achieving the following functions: 1.measuring both a curve line route and a straight line distance,generating measurement drawings, directly designating line marking anddotting; 2. adding remark and description text, images, voice and videoto the identification point; 3. performing synthetic imaging on rangingroutes and real scene photos to enhance the visualizability; 4.monitoring and controlling the status of an intelligent device on a map.These issues greatly limit the application of ranging device.

SUMMARY

In accordance with the disclosure, there is provided an interactionmethod implemented by a terminal apparatus. The method includes:obtaining an interface underlayer of a space, the interface underlayerincluding a measurement drawing having direction information anddistance information; establishing a mapping relationship between theexternal device and an identification corresponding to the externaldevice marked on the interface underlayer, the identification indicatinga physical location of the external device; receiving, from the externaldevice, property information of the external device; obtaining an imageof a real scene in the space; determining that a target object isdepicted in the image of the real scene; determining target presentationinformation of the target object according to the property informationof the external device; and displaying, on a display screencorresponding to the terminal apparatus, the image of the real scenetogether with the target presentation information, the targetpresentation information being added to the image as an augmentedreality (AR) mark at an image location corresponding to the targetobject.

In accordance with the disclosure, there is also provided a terminalapparatus including a memory and a processor coupled to the memory. Theprocessor is configured to perform: obtaining an interface underlayer ofa space, the interface underlayer including a measurement drawing havingdirection information and distance information; establishing a mappingrelationship between the external device and an identificationcorresponding to the external device marked on the interface underlayer,the identification indicating a physical location of the externaldevice; receiving, from the external device, property information of theexternal device; obtaining an image of a real scene in the space;determining that a target object is depicted in the image of the realscene; determining target presentation information of the target objectaccording to the property information of the external device; anddisplaying, on a display screen corresponding to the terminal apparatus,the image of the real scene together with the target presentationinformation, the target presentation information being added to theimage as an augmented reality (AR) mark at an image locationcorresponding to the target object.

BRIEF DESCRIPTION OF THE DRAWINGS

To more clearly illustrate the technical solution of the presentdisclosure, the accompanying drawings used in the description of thedisclosed embodiments are briefly described hereinafter. The drawingsdescribed below are merely some embodiments of the present disclosure.Other drawings may be derived from such drawings by a person withordinary skill in the art without creative efforts and may beencompassed in the present disclosure.

FIG. 1 is a schematic diagram of a forming process of an underlayer ofan intelligent interactive interface of the present disclosure;

FIG. 2 is a principal schematic diagram of an intelligent interactiveinterface of the present disclosure.

FIG. 3 is a structural diagram of an interaction process based onaugmented reality (AR) according to an example embodiment of the presentdisclosure.

FIG. 4A is an illustrative diagram showing a real scene according to anexample embodiment of the present disclosure.

FIG. 4B is an illustrative diagram showing a displayed image of realscene according to an example embodiment of the present disclosure.

FIG. 5 is an illustrative diagram showing a calibration interfaceaccording to an example embodiment of the present disclosure.

FIG. 6 is an illustrative diagram showing an image with AR markaccording to an example embodiment of the present disclosure.

FIG. 7 is an illustrative diagram showing another image with AR markaccording to an example embodiment of the present disclosure.

FIG. 8 is an illustrative diagram showing an application scenarioaccording to an example embodiment of the present disclosure.

FIG. 9 is a block diagram of an exemplary computing system according tosome embodiments of the present disclosure.

Other features, characteristics, advantages, and benefits of the presentdisclosure will become more apparent through the following detaileddescription with reference to accompanying drawings.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Embodiments of the present disclosure are described in detail below withreference to the accompanying drawings. Same or similar referencenumerals in the drawings represent the same or similar elements orelements having the same or similar functions throughout thespecification. It will be appreciated that the described embodiments aresome rather than all of the embodiments of the present disclosure. Otherembodiments obtained by those having ordinary skills in the art on thebasis of the described embodiments without inventive efforts should fallwithin the scope of the present disclosure.

Referring to FIG. 1 and FIG. 2, an intelligent interactive interface isdisclosed by the present disclosure and the intelligent interactiveinterface enables a terminal apparatus to realize functionalinteractions such as controlling, monitoring an external device. Theexternal device may be a home appliance or an industrial equipment. Theintelligent interactive interface mainly comprises an interfaceunderlayer, a plurality of identifications and a terminal apparatus. Theinterface underlayer is drawn from trajectory formed by measurement, andpreferably is an electronic map formed by performing synthetic imagingon a measurement drawing generated by ranging and a real scene photo.Thereby an electronic map is generated which implements both straightline measurement and curve line measurement, and also has many functionssuch as dotting, identification description; meanwhile, by wirelessconnecting the electronic map with an external device, the electronicmap can have the function of real-time remote monitoring, greatlyimproving the working efficiency of intelligent ranging. It isnoteworthy that, in the measurement process, the due north direction isfound as the initial measurement direction by rotating an azimuth sensordisposed on the measurement apparatus.

The measurement drawing is generated by the measurement trajectory whichis calculated according to the measured distance data and directiondata. In particular, the distance data is measured by the correspondingranging device. The ranging device may be a measurement device such asrangefinder, ranging wheel, total station instrument. The ranging devicecollects distance data as well as direction data, and transmits the datato the terminal apparatus through wireless transmission. Of course,parts of the data may also be collected by the terminal apparatus, suchas collecting direction data by a gyroscope built in cellphone. Theterminal apparatus decodes the received data according to internalprotocol, calculates and displays the data after unit conversion. Theoperations of decoding, calculating and unit conversion herein may usewell-known, commonly used methods of decoding and calculation and willno longer be discussed in detail.

If selecting the ranging wheel as the ranging device, the ranging devicemay also be used for measurement of irregular routes such as curve line,polygonal line. The ranging wheel continuously rotates, and the measureddistance data is uploaded via Bluetooth to the terminal apparatus. Theterminal apparatus receives the distance data from the ranging wheel,and meanwhile continuously obtains the current direction data of theranging wheel. The direction data may be obtained through an anglesensor, e.g. sensors such as electronic gyroscope, accelerometer. Theangle sensor may be connected to the terminal apparatus via Bluetooth ordirectly disposed inside the terminal apparatus.

After receiving the distance data and the direction data, the terminalapparatus will perform trigonometric functions calculation, based onthese two data, to obtain a moving vector of the ranging wheel as wellas a plurality of current location points. Connecting these locationpoints will form the measurement trajectory and generate thecorresponding measurement drawing at the same time. The measurementdrawing herein is formed in the ranging process, i.e., in real time. Itis not necessary to draw after the ranging process is finished. Theterminal apparatus in the present disclosure may be devices such ascellphone, tablet computer, computer.

In this embodiment, the trigonometric functions calculation formulas forthe current location points are as follows:

X=OX+sin(R)*L

Y=OY+cos(R)*L

wherein, x represents the X-axis coordinate of the current locationpoint, Y represents the Y-axis coordinate of the current location point,OX is the X-axis coordinate of last point, OY is the Y-axis coordinateof last point, R is the directional angle value when obtaining thedistance value, L is the distance difference value of the currentlocation point and the last point, (X, Y) is the calculated location ofthe current location point.

Then the measurement drawing and the real scene photo of the measuredtarget are synthesized to form the interface underlayer, i.e., anelectronic map, which can improve the visualizability of the measurementtrajectory. The specific synthesis process is as follows: take a realscene photo of the measured target; then measure the measured targetusing the ranging device of the present disclosure to obtain themeasurement trajectory; and generate a corresponding measurement drawingor directly use an already obtained measurement drawing; overlap andsynthesize this measurement trajectory and the real scene photo. In thesynthesis process, parameters such as the perspective angle, perspectivedistance, direction of the trajectory are adjusted to make thetrajectory aligned and attached to the measured target. Of course, theinterface underlayer may also be directly formed by the measurementdrawing which is formed by surveying and mapping without synthesizingwith a real scene photo.

In the measurement process, when it is required to record the locationof the external device, an identification may be added to the currentlocation of the generated measurement drawing. Each identificationcorresponds to an external device at the current location. If it isfurther required to add corresponding identification information to theidentification, then this identification is selected for which to add anidentification information. The identification information is mainly theproperty information of the external device.

The external device is connected to the terminal apparatus in a wired orwireless way. Interactive control of the external device may beimplemented by the terminal apparatus. The property information andworking data of the external device is uploaded and stored in a serverand displayed through the interface underlayer. That is, a one-to-onemapping relationship between the information of the external device(including the property information and working data) and the externaldevice is established. As such, when a cursor of the terminal apparatuspauses on or clicks the identification, the information of thecorresponding external device is displayed on the terminal apparatus.The identification as well as the identification informationcorresponding to the identification on the measurement drawing can beedited, moved or deleted. Generally speaking, the terminal apparatus isconnected to the external device and is used to display the interfaceunderlayer, identification and control and/or exchange information withthe external device.

The identification information herein is not limited to the format oftext, picture, video or voice and the identification and information isuploaded in real time and stored in the internal database of theterminal apparatus for later viewing and use at any time. Thus the addedidentification information can be newly added, or can be read directlyfrom the database for use. For instance, the added picture can be newlytaken photo or can also be selected from the existing album of theterminal apparatus. The newly added identification point andidentification information will also be stored in the database in realtime for convenient call.

When the information (i.e., the identification information herein) ofthe external device is abnormal, especially the working data, theintelligent interactive interface of the present disclosure will giveout alarm information and the alarm information is given out in form ofvoice or light flickering through the terminal apparatus.

Under network environment, the identification point and itscorresponding identification information can both synchronized to acloud server for permanent storage and on-demand distribution. As such,it is ensured that the identification data is not prone to loss, and canbe called at any time conveniently.

In addition, the present disclosure may also pause the measurement inthe measurement process, and continue the measurement after the locationof the ranging device is adjusted, which can control the ranging processvery well. In practice, the pause function is implemented by apause/continue button disposed on the ranging device or terminalapparatus. When the measurement is paused, any one of the previousidentifications may also be re-selected as a new starting point formeasurement. Any one of existing generated measurement drawings may alsobe opened and the measurement is continued from the end of lastmeasurement, or any one of previous identifications is re-selected as anew starting point for measurement. Controlling the ranging processaccording to the user requirement is thus accomplished, greatlyimproving the flexibility of the ranging process.

The present disclosure provides an intelligent interactive interface,comprising: an interface underlayer drawn from trajectory formed bymeasurement; a plurality of identifications disposed on the interfaceunderlayer, each of the identifications corresponds to an externaldevice, information of the external device is uploaded in real time,displayed on the interface underlayer and capable of being stored on aserver, and a mapping relationship being established between theinformation of the external device and the corresponding identificationof respective external device; a terminal apparatus to connect to theexternal device and display the interface underlayer, theidentifications, and control the external device and/or exchangeinformation with the external device; wherein the information of theexternal device is displayed on the terminal apparatus, in real time,through the identifications and the identifications can be added ordeleted in real time.

In some embodiments, the interface underlayer is an electronic mapformed by performing synthetic imaging on measurement drawing generatedby ranging and a real scene photo; or the interface underlayer isdirectly formed by the measurement drawing generated by ranging.

In some embodiments, the measurement drawing is formed in real time in ameasurement process.

In some embodiments, trigonometric functions calculation is madeaccording to the measured distance data and direction data to obtain aplurality of location points and connecting the plurality of locationpoints forms the measurement drawing.

In some embodiments, when a cursor of the terminal apparatus is pausedon the identification or clicking the identification point, theinformation of the corresponding external device is displayed on theterminal apparatus.

In some embodiments, trigonometric functions calculation formulas of thecurrent location point are:

X=OX+sin(R)*L

Y=OY+cos(R)*L

wherein X represents the X-axis coordinate of the current locationpoint, Y represents the Y-axis coordinate of the current location point,OX is the X-axis coordinate of the last point, OY is the Y-axiscoordinate of the last point, R is the directional angle value whenobtaining the distance data, L is a difference value of the distancesbetween the current location point and the last point.

In some embodiments, in the measurement process, the identification ismarked at the location on the generated measurement drawingcorresponding to the external device and identification information isadded in the server for the identification.

In some embodiments, the identification information comprises any one ofpicture, text, voice, and video.

In some embodiments, in the ranging process, the measurement is pausedor any one of the previous identifications is re-selected as a newstarting point for measurement.

In some embodiments, any one of existing generated measurement drawingsis opened and the measurement is continued from the end of lastmeasurement.

In some embodiments, the intelligent interactive interface gives outalarm information when the information of the external device isabnormal.

In some embodiments, in the process of synthesizing the measurementdrawing and the real scene photo of a measured target, the perspectiveangle, the perspective distance and direction of the measurement drawingare adjusted to make the measurement drawing aligned and attached to themeasured target.

In some embodiments, the information of the external device comprisesthe inherent property and working data of the external device.

In some embodiments, in the measurement process, the due north directionis found as the initial measurement direction by rotating an azimuthsensor disposed on the measurement apparatus.

The beneficial effects of some embodiments of the present disclosureare: 1. The disclosed method and device can measure the distance ofvarious routes such as straight line, curve line and polygonal line. 2.The disclosed method and device can generate measurement drawings inreal time according to actual measurement, and can direct the linemarking, dotting according to the measurement drawing, or addidentification descriptions to identification points in form of, but notlimited to, picture, text, video or voice. 3. The disclosed method anddevice can perform synthetic imaging on a measurement drawing and a realscene photo to enhance the visualizability. 4. The disclosed method anddevice can perform remote real-time monitor and control of an electronicmap and thus greatly improve its working efficiency.

The present disclosure also provides an interaction method based onaugmented reality (AR). The method can be implemented by a terminalapparatus. As shown in FIG. 3, the terminal apparatus may obtain aninterface underlayer of a space (S302). The interface underlayerincludes a measurement drawing having direction information and distanceinformation. The measurement drawing may be generated according tomeasurement of a 3D space (e.g., and objects in the 3D space) performedby a ranging device. In one embodiment, the terminal apparatus may beconnected to the ranging device and generate the interface underlayeraccording to the measurement drawing. In another embodiment, theterminal apparatus may obtain the measurement drawing from a file storedlocally or transmitted online.

At least one external device may be marked on the interface underlayer.The terminal apparatus can establish a mapping relationship between anexternal device and an identification corresponding to the externaldevice marked on the interface underlayer (S304). The identification onthe interface underlayer indicates a physical location of the externaldevice. The procedure of making the identification and establishing themapping relationship can be similar to the process described in previousembodiments and is not repeated herein.

The terminal apparatus and the external device can be connected and theterminal apparatus can receive information from the external device(S306). Information exchange can be performed between the terminalapparatus and the external device when they are connected. The externaldevice may transmit various types of property information to theterminal apparatus, such as characteristics information, current statusinformation, record information, control options, etc. Thecharacteristics information can indicate one or more general features ofthe external device that does not change often, such as name, model,dimensions, operation capacity, and other characteristics relating togeneral functions of the external device. In some embodiments, if theexternal device corresponds to another object of interest, thecharacteristics information may further include features of the objectof interest, such as name, type, model, etc. The status information mayindicate current operation status of the external device, such as on/offstatus, battery level, volume, brightness, status measured by sensor(s)on the external device, and other status relating to functions of theexternal device. The record information may include any record kept bythe external device, such as operation log, historical record, savedmedium object (e.g., picture, text, audio, video), etc. The controloptions may include options available to be used by the terminalapparatus for controlling the external device, such as read data (e.g.,navigate a control menu, query a record or a status), write data (e.g.,adjust an operation parameter), etc. The terminal apparatus may send acontrol instruction to the external device. The control instruction maybe in correspondence with the control options provided by the externaldevice. When receiving the control instruction, the external device isconfigured to follow the control instruction and perform correspondingactions, such as sending read data to the terminal apparatus, write datareceived from the terminal apparatus to the external device, adjustoperation parameter and send adjusting result (e.g., success/fail, oradjusting result with corresponding operation details), provide anupper/lower control menu items based on navigation control instruction,etc.

The terminal apparatus obtains an image of a real scene in the space(S308). The image of the real scene may be obtained through any properimaging device associated with the terminal apparatus, such as ARglasses and/or a camera embedded in the terminal apparatus. The image ofthe real scene may be captured and obtained in real time. The image ofthe real scene may describe and include one or more objects. The one ormore objects may include a target object. The target object may be theexternal device, or an object of user interest corresponding to theexternal device. The object of user interest may not have capability toestablish communication and interact with the terminal apparatus. Forexample, a smart soil sensor having communication capabilities can beinserted to a pot having a plant. The plant can be the object ofinterest, and the smart soil sensor can be the external devicecorresponding to the object of interest. The image of the real scene maybe displayed at a display device associated with the terminal apparatusin real time, such as a display screen of the terminal apparatus, ARglasses, etc. When the imaging device is moved and/or rotated, the imageof the real scene may be updated accordingly. For example, a room shownin FIG. 4A can be captured and displayed as an image through AR glasses102 as shown in FIG. 4B. The image may include objects such as externaldevice 200A (a cleaning robot), external device 200B (a soil sensor), aplant 402, a door 404, and a clock 406. FIG. 8 shows another examplewhere an image of a corner of the room is shown on a terminal apparatus100.

Returning to FIG. 3, the terminal apparatus may determine that a targetobject is depicted in the image of the real scene (S310). For example,the interface underlayer of the space shown in FIG. 4 may include atleast two external devices: a cleaning robot 200A and a soil sensor200B. It can be determined that two target objects are depicted in theimage of the real scene shown in FIG. 4: the cleaning robot 200A, andthe plant 402 corresponding to the soil sensor 200B.

In some embodiments, S310 may further include performing a calibrationprocess and determining a location and an orientation of the terminalapparatus in the interface underlayer; determining a target location ofthe external device relative to the terminal apparatus according to thelocation of the terminal apparatus in the interface underlayer and theidentification of the external device in the interface underlayer; anddetermining that the external device is in the image of the real sceneif the orientation of the terminal apparatus indicates that the terminalapparatus is facing toward the external device.

The calibration process may start when the terminal apparatus activatesAR function. For example, when showing the image of the real scene, theterminal apparatus may automatically activate the AR function or promptthe user to activate the AR function. Upon receiving an AR activationinstruction, the terminal apparatus may perform the calibration processto determine a location and an orientation of the imaging device of theterminal apparatus on the interface underlayer. The terminal apparatusmay display instructions and guide the user to complete the calibrationprocess. FIG. 5 is an illustrative diagram showing a calibrationinterface according to an example embodiment of the present disclosure.For example, as shown in FIG. 5, a calibration icon 502B may be added tothe real time image shown on the display device 102 of the terminalapparatus at a preset image location, such as added to a center of thereal time image. The calibration icon indicates an expected outline of acalibration object 406 to be found. The terminal apparatus may alsodisplay a guidance message 502A to inform the user to find and point tothe calibration object in the real scene so that the calibration objectshown in the image of the real scene match the expected outline. Thecalibration object has a corresponding location and orientation in theinterface underlayer. Said correspondence can be established during theranging process and/or according to the measurement drawing. The usercan move and point the imaging device at the calibration object in thereal scene to make the calibration object shown in the real time imagematch the calibration icon. Upon determining that the calibration objectincluded in the image of the real scene matches the expected outline, itis determined that terminal apparatus is located at a reference pointwith a reference orientation on the interface underlayer.

It can be understood that the calibration process is performed under theassumption that the location of the calibration object remains at thereference point since the time of generating the measurement drawing.

In one embodiment, the user may manually confirm that the calibrationobject shown in the real time image matches the calibration icon. Inanother embodiment, the terminal apparatus may determine whether thecalibration object included in the image of the real scene matches theexpected outline (i.e., the calibration icon) according to the image ofthe real scene and prestored image data corresponding to the calibrationobject. In one example, the calibration object matching the expectedoutline may refer to that the calibration object is shown at the presetimage location with an outline overlapping the expected outline. Inanother example, the terminal apparatus can compare image data in anarea indicated by the calibration icon (e.g., an area covered by theexpected outline or an area including and greater than the area coveredby the expected outline) with the prestored image data of thecalibration object to obtain a comparison result, and the calibrationobject matching the expected outline may refer to that a similaritybetween the image data of the area indicated by the calibration icon andthe prestored image data is greater than a similarity threshold.

If the comparison result indicates that the calibration object is nearbybut not yet completely match with the calibration icon (e.g., part ofthe area indicated by the calibration icon matches part of the prestoredimage data), the terminal apparatus may determine a distance and adirection of the calibration object relative to the terminal apparatusaccording to the real time image, and determine the location of theterminal apparatus in the interface underlayer accordingly.Alternatively, the terminal apparatus may provide movement instructionsaccording to the matching parts, such as move closer/further away, turnto right/left, point higher/lower, so that the user can move the imagingdevice to make the calibration object accurately match the calibrationicon.

Upon determining that the calibration object included in the image ofthe real scene matches the expected outline, it is determined thatterminal apparatus is located at a reference point with a referenceorientation on the interface underlayer. In some embodiments, a locationof the reference point and the reference orientation, the calibrationicon, and the preset image location are recorded during a process ofgenerating the measurement drawing with a ranging device. For example,when the calibration object matches with the expected outline, adistance and a direction of the calibration object relative to theterminal apparatus is determined as the location and orientationcorresponding to the calibration icon. Further, a distance and relativeorientation between the calibration object and a reference point in themeasurement drawing are known according to their 3D coordinates. Duringranging, an image of the calibration object is taken by a camera locatedat the reference point and oriented at a reference orientation. Theorientation and location of the reference point are recorded. The imageof the calibration object can be used to generate the calibration iconand record the preset image location. During calibration, when thecalibration object matches the calibration icon on the real time image,it is determined that the imaging device of the terminal apparatus ismoved to the location of the reference point with the referenceorientation. In this way, the location and orientation of the imagingdevice of the terminal apparatus on the interface underlayer isdetermined to be the same as the recorded location and orientation ofthe reference point.

In some embodiments, an expected image location of the external devicein the image of the real scene can be determined according to theidentification (physical location) of the external device in theinterface underlayer and the location and the orientation of theterminal apparatus in the interface underlayer. For example, when thecalibration process is completed, the terminal apparatus can determineexpected location and orientation of any objects marked in the interfaceunderlayer. The terminal apparatus may be moved after the calibrationprocess and the image of the real scene may be updated in real time.Further, the terminal apparatus may update its location and orientationon the interface underlayer in real time based on its movement. In someembodiments, the target object may be an object marked in the interfaceunderlayer. The terminal apparatus can determine whether an objectmarked in the interface underlayer is expected to be depicted in theimage of the real scene according to the location and orientation of theterminal apparatus on the interface underlayer and the location of theobject in the interface underlayer. Here, “expected to be depicted”refers to an assumption that the location of the object has not changedsince the interface underlayer is obtained. If the physical location ofthe object is changed, the location of the object marked on theinterface underlayer is incorrect, and the object may not appear in theimage of the real scene when the terminal apparatus is facing the markedlocation. In some embodiments, the terminal apparatus may display anobject marked in the interface underlayer but occluded in the image ofreal scene (e.g., by overlaying said object in the image of the realscene). For example, the image of the real scene shown in FIG. 4 doesnot show details inside the bedroom on the right side because suchdetails are blocked by walls of the room. The interface shown in FIG. 6can include layout 608 of the bedroom according to the interfaceunderlayer.

In some embodiments, the terminal apparatus may compare current imagedata at the expected image location of the image of the real scene withprestored image data of the external device; and upon determining thatthe current image data does not match the prestored image data,displaying a reminder message indicating that a current location of theexternal device in the real scene is not consistent with the physicallocation of the external device recorded in the interface underlayer.For example, as shown in FIG. 8, a table 802′ in the real scene is shownin the image on the terminal apparatus 100 as table 802. An externaldevice 200C (a laptop) expected to be on the table 802 according to itsidentification in the interface underlayer is not actually on the table.This situation may occur when the interface underlayer is a designlayout and the target object is not yet placed to the expected locationyet. In this case, the terminal apparatus may guide the user to move theexternal device so that the external device occurs at the expected imagelocation, to implement the design layout. In another example, theexternal device is desired to be moved away from its previous location,and the terminal apparatus may guide the user to point the terminalapparatus to the new location of the external device and update thephysical location of the external device in the interface underlayeraccording to the new location. The guidance provided by the terminalapparatus may be displayed as AR mark added to the image of the realscene as well.

In some embodiments, the terminal apparatus may determine whether atarget object is depicted in the image of the real scene by comparingthe image of the real scene with prestored image/model datacorresponding to the target object and determining whether there is anarea/section in the image of the real scene that is similar to theprestored image/model data. For example, object recognition may beperformed on the image of the real scene to determine edges/outlines ofcandidate objects in the image. When comparing with each candidateobject, the prestored image/model data may be resized, rotated and/orcropped according to the edges/outlines of the candidate object. Otherimage features may also be used for comparison such as colorinformation.

In some embodiments, the terminal apparatus may determine that a targetobject is depicted in the image of the real scene according to userselection. For example, the user may select an object in the image ofthe real scene as the target object.

In some embodiments, the location and size of the target object in theimage of the real scene is determined. In cases where the target objectis automatically determined by the terminal apparatus (e.g., byactivating the AR function or comparing image with prestored objectimage data), the location and the size of the target object is alreadydetermined in the automatic determination process. In cases where thetarget object is manually selected by a user, edge detection may beperformed in the image area selected by the user, and an outline of thetarget object may be obtained according to the result of edge detection,as well as its location and size information. In some embodiments, aframe equal to or slightly bigger than a bounding box containing thetarget object at its center may be used to represent the location andsize of the target object in the interface underlayer.

Returning to FIG. 3, the terminal apparatus may determine targetpresentation information of the target object according to the propertyinformation of the external device (S312). When the target object is theexternal device, the target presentation information may be obtaineddirectly from the property information of the external device. When thetarget object is not an external device, the terminal apparatus maydetermine a target external device corresponding to the target object,and determine the target presentation information according to theproperty information of the target external device. As used herein, atarget object that is not an external device but has a correspondingexternal device may be referred to as an object of interest. As usedherein, an external device may refer to the external device identifiedas the target object and/or the external device corresponding to theobject of interest identified as the target object. In one embodiment,the object of interest also has a mapping relationship with anidentification on the interface underlayer. In another embodiment, whenthe object of interest is not yet marked in the interface underlayer,the mapping relationship can be established according to the relativelocation between the terminal apparatus and the object of interest(e.g., determined according to the image of the real scene), and thelocation and orientation of the terminal apparatus on the interfaceunderlayer.

The terminal apparatus displays, on a display screen corresponding tothe terminal apparatus, the image of the real scene together with thetarget presentation information (S314). The target presentationinformation is added to the image as an augmented reality (AR) mark atan image location corresponding to the target object. The AR mark caninclude one or more of an image, a text, a video. In some embodiments,the AR mark may further include an audio that can be played when thetarget object is determined and/or upon receiving a user instruction toplay audio related to the target object. The audio may be a voiceconverted from the displayed text, and/or a sound effect related to thedisplayed text/image/video. FIGS. 6-7 shows some interface examples ofAR mark being added/overlaid to the image of the real scene shown inFIG. 4.

The target presentation information and the corresponding AR mark caninclude various information related to the target object. In someembodiments, the target presentation information includes identity ofthe target object. The identity may be a result from object recognitionof the object of interest, or a name/model of the external deviceobtained from the characteristics information or the interfaceunderlayer. The AR mark may show, for example, “flower”, “table”, “appletree Row 3 Column 5”, “light switch 2”, “fridge”, “living room light”,“robot vacuum”. The AR marks 602A and 604A shown in FIGS. 6-7 are bothexamples of characteristics information of target objects.

In some embodiments, the target presentation information may indicatelocation and/or size of the target object. The location and/or sizeinformation can be determined according to the measurement drawingand/or the image of the real scene. The location may be described by adistance between the target object and another object (e.g., theterminal apparatus, an external device, the reference point used incalibration, etc.). The size may be described by dimension information,such as length, width, height, volume of the target object or thebounding box of the target object. Correspondingly, the AR mark canshow, for example, “1 meter from you”, “2 meter from the door”, “height:30 cm”, etc.

In some embodiments, the target presentation information may includestatus information and/or record information of the external deviceobtained through the communication between the terminal apparatus andthe external device. For example, the AR mark for a target object mayshow: “battery level: 80%”, “freezer temp: 4° F.”, “status: heating”,“soil humidity: dry, time to add water”, “up and running for 30 hours 20minutes”. The AR marks 602B and 604B shown in FIGS. 6-7 are bothexamples of status information of target objects.

In some embodiments, the target presentation information may includecontrol options of the external device obtained through thecommunication between the terminal apparatus and the external device.The AR mark may be used as user interface (UI) item to facilitate theuser to interact with the external device through the terminalapparatus. The UI item or sub-items shown as the AR mark may correspondto control options provided by the external device. For example, someexamples of UI item and sub-item used as the AR mark may include: “checkoperation status”, “change name”, “return to main menu”, “adjustvolume”, “search operation log”, a text input box, etc. The AR marks602C through 602H shown in FIGS. 6-7 are all examples of UI itemscorresponding to the target object 200A.

In some embodiments, after displaying the AR mark, the terminalapparatus may generate a control instruction upon detecting a useroperation on the AR mark; and send the control instruction to theexternal device, the control instruction being configured to instructthe external device to perform a corresponding action.

In some embodiments, after sending the control instruction, the terminalapparatus may receive a feedback information corresponding to thecontrol instruction; update the target presentation informationaccording to the feedback message; and display the updated targetpresentation information as an updated AR mark.

In some embodiments, when the control instruction corresponds to a writeaction of adjusting a parameter of the external device, the feedbackmessage includes an execution result indicating whether thecorresponding action is executed successfully; and when the controlinstruction corresponds to a read action of querying a record of theexternal device, the feedback message includes a query result.

In some embodiments, when the AR mark is displayed as one or more userinterface (UI) items corresponding to one or more control options of theexternal device, and generating the control instruction includes: upondetecting that the user operation is performed on a UI item having oneor more sub-items, updating the AR mark to display the one or moresub-items; and upon detecting that the user operation is performed on aUI item or a sub-item corresponding to a control option of the externaldevice, generating the control instruction according to the useroperation and the control option. That is, the AR mark may be used asuser interface (UI) to interact with the external device. The AR markmay display icons and/or menu items indicating control options of theexternal device, and upon detecting user operation on a displayed iconor menu item, the terminal apparatus may perform an action correspondingto the icon or menu item. The action may include sending a controlinstruction to the external device, and/or displaying one or moreadditional control options corresponding to the selected icon or menuitem. The action may further include receiving a feedback of the controlinstruction from the external device. For example, the AR mark caninclude an on/off toggle icon; and when the toggle icon is selected, theterminal apparatus instructs the external device to switch on/offstatus. In another example, the AR mark can include a volume icon; andwhen the volume icon is selected, an additional control option, i.e., avolume adjusting bar, is displayed as the AR mark. Upon detecting useroperation on the volume adjusting bar, the terminal apparatus sends acontrol instruction to adjust a volume of the external device. Inanother example, the AR mark can include main menu items inconfiguration setting panel of the external device; and when a main menuitem is selected, one or more sub menu items can be displayed as the ARmark. In another example, the AR mark may include a text box or an audioinput icon to collect user-inputted information. The user-inputtedinformation may be a search query or a voice control command. Theterminal apparatus may instruct the external device to respond to thesearch query; or recognize a control instruction from the voice controlcommand and instruct the external device to follow the controlinstruction. In another example, the terminal apparatus may receive afeedback from the external device, such as a message indicating successor failure of executing the control instruction, a search resultcorresponding to the search query, etc. By using the AR mark as controlUI, the user can directly control the external device through operationson the AR mark without the need to open a specific control applicationor use a remote to control the external device.

As shown in FIGS. 6 and 7, the AR mark 602C is a UI item representing anon/off switch and by detecting user operation on 602C, the terminalapparatus can directly instruct the cleaning robot 200A to turn on orturn off. The AR mark 602D is a UI item representing a configurationmenu, and upon detecting 602D being selected, AR marks 602E, 602F and602H representing sub-items may be displayed. Further, upon detecting ARmark 602E “show cleaning route” being selected, 602E may be highlighted,and information about the cleaning route may be obtained from thecleaning robot 200A and rendered as AR mark 602G shown in FIG. 7. Insome embodiments, according to the user selection of showing thecleaning route, AR marks 604A and 604B may be hidden or removed fromdisplay when showing AR mark 602G. In addition, when AR mark 602Fshowing “mark travel stop” is selected, the terminal apparatus maydetect a user selection at an image location as a stop destination,determine a physical location of the stop destination corresponding tothe image location in the interface underlayer, and inform the cleaningrobot about the coordinates of the stop destination so that the cleaningrobot can update the cleaning route by adding the stop destination. ARmark 602H is a sub-item configured to return to a previous UI item(e.g., mark 602D) upon selection.

In some embodiments, besides current operation status, the targetpresentation information can provide additional insights in the timedimension. For example, the terminal apparatus may establish mappingrelationship with a plurality of external devices located at differentsections/areas of the 3D space corresponding to the interfaceunderlayer. When one external device appears in the image of the realscene, the AR mark corresponding to the external device can show both amalfunction status and a time when the malfunction happens, such as“error occurred 30 minutes ago ”. The AR mark may also show operationstatus of itself and a group that the external device belongs to, suchas “status: normal, number of malfunction devices in my group: 3.”

In some embodiments, after showing the AR mark of a target object, theterminal apparatus may receive a user instruction of providing adistance between the target object and another object. For example, asshown in FIG. 6, the target object may be 200A, the other objectselected by user (e.g., shown as 610) may be the door. AR mark 606 showsthe distance between the cleaning robot and the door. If the otherobject is marked in the interface underlayer, the terminal apparatus candirectly determine a first vector between the target object and theother object according to their coordinates in the interface underlayer.If the other object is not marked in the interface underlayer, theterminal apparatus may determine a second vector between the otherobject and itself through its measurement function and/or the image ofthe real scene, and determine the distance between the other object andthe target object according to the second vector and a third vector fromthe target object to the terminal apparatus.

The terminal apparatus is configured to determine a location of anyobject in a captured image relative to itself (e.g., depth information,orientation information of the object) by using any proper measurementtechniques, such as using image data of two or more cameras, rangingdata from Lidar sensor, using data from other sensors such as locationsensor, proximity sensor, inertial sensor, field sensor, or acombination thereof.

The terminal apparatus is configured to detect a user selection on theimage of the real scene. The user selection may be a touch operation ona touch screen of the terminal apparatus, a voice selection captured byan audio device of the terminal apparatus, a hand/body gesturerecognized by the terminal apparatus, etc.

The terminal apparatus is associated with a display device and animaging device. In some embodiments, the display device and/or theimaging device may be embedded in the terminal apparatus. In someembodiments, the display device and/or the imaging device may be aseparate device connected to the terminal apparatus. It can beunderstood that certain embodiments are described based on an assumptionthat the location of the terminal apparatus is the same as the imagingdevice. When describing a relative location between the terminalapparatus and an object according to the image of the real scene, it maybe interpreted as a distance between the imaging device of the terminalapparatus and the object.

FIG. 9 is a block diagram of an exemplary computing system/devicecapable of implementing the disclosed interaction method according tosome embodiments of the present disclosure. As shown in FIG. 9,computing system 900 may include a processor 902 and a storage medium904. According to certain embodiments, the computing system 900 mayfurther include a display 906, a communication module 908, additionalperipheral devices 992, and one or more bus 994 to couple the devicestogether. Certain devices may be omitted and other devices may beincluded. The terminal apparatus and/or the external device may beimplemented on the computing system 900.

Processor 902 may include any appropriate processor(s). In certainembodiments, processor 902 may include multiple cores for multi-threador parallel processing, and/or graphics processing unit (GPU). Processor902 may execute sequences of computer program instructions to performvarious processes, such as measuring a distance to a nearby object,exchanging information with another device, displaying AR marks, etc.Storage medium 904 may be a non-transitory computer-readable storagemedium, and may include memory modules, such as ROM, RAM, flash memorymodules, and erasable and rewritable memory, and mass storages, such asCD-ROM, U-disk, and hard disk, etc. Storage medium 904 may storecomputer programs for implementing various processes, when executed byprocessor 902. Storage medium 904 may also include one or more databasesfor storing certain data such as measurement drawings, image and/ormodel data of certain objects, and certain operations can be performedon the stored data, such as database searching and data retrieving.

The communication module 908 may include network devices forestablishing connections through a network. Display 906 may include anyappropriate type of computer display device or electronic device display(e.g., AR glasses, touch screens). Peripherals 992 may includeadditional I/O devices, such as a controller, a microphone, and so on.

In operation, the processor 902 may be configured to executeinstructions stored on the storage medium 904 and perform variousoperations related to an interaction method as detailed in the previousdescriptions.

Although various embodiments of the present disclosure have beendescribed, it is apparent to those skilled in the art that variouschanges and modifications can be made without departing from the spiritand scope of the present disclosure to achieve one or more advantages ofthe present disclosure. For those skilled in the art, one or morecomponents may be replaced by other components performing the identicalfunctions. It should be understood that the features described hereinwith reference to a particular drawing can be combined with anotherfeature in another drawing, even if such a case is not explicitlymentioned. In addition, the method of present disclosure may beimplemented all by software being executed by a processor or may beimplemented in a hybrid manner by a combination of hardware logic andsoftware logic to achieve the same result. Such modifications to theembodiments of the present disclosure are intended to be covered by theappended claims.

What is claimed is:
 1. An interaction method, implemented by a terminalapparatus, comprising: obtaining an interface underlayer of a space, theinterface underlayer including a measurement drawing having directioninformation and distance information; establishing a mappingrelationship between an external device and an identificationcorresponding to the external device marked on the interface underlayer,the identification indicating a physical location of the externaldevice; receiving, from the external device, property information of theexternal device; obtaining an image of a real scene in the space;determining that a target object is depicted in the image of the realscene; determining target presentation information of the target objectaccording to the property information of the external device; anddisplaying, on a display screen corresponding to the terminal apparatus,the image of the real scene together with the target presentationinformation, the target presentation information being added to theimage as an augmented reality (AR) mark at an image locationcorresponding to the target object.
 2. The method according to claim 1,wherein the target object is the external device or an object ofinterest corresponding to the external device.
 3. The method accordingto claim 1, wherein: the property information of the external deviceincludes at least one of characteristics information of the externaldevice, current status information of the external device, recordinformation on the external device, or control options of the externaldevice.
 4. The method according to claim 1, after displaying the ARmark, further comprising: generating a control instruction upondetecting a user operation on the AR mark; and sending the controlinstruction to the external device, the control instruction beingconfigured to instruct the external device to perform a correspondingaction.
 5. The method according to claim 4, wherein the AR mark isdisplayed as one or more user interface (UI) items corresponding to oneor more control options of the external device, and generating thecontrol instruction comprises: upon detecting that the user operation isperformed on a UI item having one or more sub-items, updating the ARmark to display the one or more sub-items; and upon detecting that theuser operation is performed on a UI item or a sub-item corresponding toa control option of the external device, generating the controlinstruction according to the user operation and the control option. 6.The method according to claim 4, after sending the control instruction,further comprising: receiving a feedback information corresponding tothe control instruction; updating the target presentation informationaccording to the feedback message; and displaying the updated targetpresentation information as an updated AR mark.
 7. The method accordingto claim 6, wherein: when the control instruction corresponds to a writeaction of adjusting a parameter of the external device, the feedbackmessage includes an execution result indicating whether thecorresponding action is executed successfully; and when the controlinstruction corresponds to a read action of querying a record of theexternal device, the feedback message includes a query result.
 8. Themethod according to claim 1, wherein the target object is the externaldevice, and determining that the target object is depicted in the imageof the real scene comprises: performing a calibration process anddetermining a location and an orientation of the terminal apparatus inthe interface underlayer; determining a target location of the externaldevice relative to the terminal apparatus according to the location ofthe terminal apparatus in the interface underlayer and theidentification of the external device in the interface underlayer; anddetermining that the external device is in the image of the real sceneif the orientation of the terminal apparatus indicates that the terminalapparatus is facing toward the external device.
 9. The method accordingto claim 8, wherein performing the calibration process comprises:displaying a calibration icon on the image of the real scene at a presetimage location, the calibration icon indicating an expected outline of acalibration object; and upon determining that the calibration objectincluded in the image of the real scene matches the expected outline,determining that terminal apparatus is located at a reference point witha reference orientation on the interface underlayer.
 10. The methodaccording to claim 9, wherein: a location of the reference point and thereference orientation, the calibration icon, and the preset imagelocation are recorded during a process of generating the measurementdrawing with a ranging device.
 11. The method according to claim 8,further comprising: determining an expected image location of theexternal device in the image of the real scene according to theidentification of the external device in the interface underlayer andthe location and the orientation of the terminal apparatus in theinterface underlayer.
 12. The method according to claim 11, wherein:comparing current image data at the expected image location of the imageof the real scene with prestored image data of the external device; andupon determining that the current image data does not match theprestored image data, displaying a reminder message indicating that acurrent location of the external device in the real scene is notconsistent with the physical location of the external device recorded inthe interface underlayer.
 13. A terminal apparatus, comprising: amemory, and a processor coupled to the memory and configured to perform:obtaining an interface underlayer of a space, the interface underlayerincluding a measurement drawing having direction information anddistance information; establishing a mapping relationship between theexternal device and an identification corresponding to the externaldevice marked on the interface underlayer, the identification indicatinga physical location of the external device; receiving, from the externaldevice, property information of the external device; obtaining an imageof a real scene in the space; determining that a target object isdepicted in the image of the real scene; determining target presentationinformation of the target object according to the property informationof the external device; and displaying, on a display screencorresponding to the terminal apparatus, the image of the real scenetogether with the target presentation information, the targetpresentation information being added to the image as an augmentedreality (AR) mark at an image location corresponding to the targetobject.
 14. The terminal apparatus according to claim 13, wherein thetarget object is the external device or an object of interestcorresponding to the external device.
 15. The terminal apparatusaccording to claim 13, wherein: the property information of the externaldevice includes at least one of characteristics information of theexternal device, current status information of the external device,record information on the external device, or control options of theexternal device.
 16. The terminal apparatus according to claim 13,wherein after displaying the AR mark, the processor is furtherconfigured to perform: generating a control instruction upon detecting auser operation on the AR mark; and sending the control instruction tothe external device, the control instruction being configured toinstruct the external device to perform a corresponding action.
 17. Theterminal apparatus according to claim 16, wherein the AR mark isdisplayed as one or more user interface (UI) items corresponding to oneor more control options of the external device, and generating thecontrol instruction comprises: upon detecting that the user operation isperformed on a UI item having one or more sub-items, updating the ARmark to display the one or more sub-items; and upon detecting that theuser operation is performed on a UI item or a sub-item corresponding toa control option of the external device, generating the controlinstruction according to the user operation and the control option. 18.The terminal apparatus according to claim 16, wherein after sending thecontrol instruction, the processor is further configured to perform:receiving a feedback information corresponding to the controlinstruction; updating the target presentation information according tothe feedback message; and displaying the updated target presentationinformation as an updated AR mark.
 19. The terminal apparatus accordingto claim 18, wherein: when the control instruction corresponds to awrite action of adjusting a parameter of the external device, thefeedback message includes an execution result indicating whether thecorresponding action is executed successfully; and when the controlinstruction corresponds to a read action of querying a record of theexternal device, the feedback message includes a query result.
 20. Theterminal apparatus according to claim 13, wherein the target object isthe external device, and determining that the target object is depictedin the image of the real scene comprises: performing a calibrationprocess and determining a location and an orientation of the terminalapparatus in the interface underlayer; determining a target location ofthe external device relative to the terminal apparatus according to thelocation of the terminal apparatus in the interface underlayer and theidentification of the external device in the interface underlayer; anddetermining that the external device is in the image of the real sceneif the orientation of the terminal apparatus indicates that the terminalapparatus is facing toward the external device.